Sensor calibration system, display control apparatus, program, and sensor calibration method

ABSTRACT

A sensor calibration system is provided that includes a sensor apparatus including an event-driven vision sensor including a sensor array configured with sensors that generate event signals upon detection of a change in incident light intensity, and a display apparatus including a display section configured to change luminance of a planar region instantaneously with a predetermined spatial resolution as per a calibration pattern of the sensors.

TECHNICAL FIELD

The present invention relates to a sensor calibration system, a displaycontrol apparatus, a program, and a sensor calibration method.

BACKGROUND ART

There are known event-driven vision sensors that cause pixels detectingchanges in incident light intensity to generate signals in atime-asynchronous manner. The event-driven vision sensor is moreadvantageous than frame-type vision sensors that scan all pixels atpredetermined intervals, specifically as in the case of CCD(Charge-Coupled Device) or CMOS (Complementary Metal-OxideSemiconductor) image sensors, due to its ability to run at high speed onlow power. The technologies related to the event-driven vision sensorsare described in PTLs 1 and 2, for example.

CITATION LIST Patent Literature

[PTL 1] JP 2014-535098T [PTL 2] JP 2018-85725A

SUMMARY Technical Problem

However, although the above mentioned advantages of the event-drivenvision sensors are well known, peripheral technologies that take intoconsideration their characteristics different from those of frame-typevision sensors, for example, have yet to be proposed sufficiently.

In view of the above, the present invention is aimed at providing asensor calibration system, a display control apparatus, a program, and asensor calibration method for efficiently calibrating event-drivenvision sensors.

Solution to Problem

According to one aspect of the present invention, there is provided asensor calibration system including: a sensor apparatus including anevent-driven vision sensor including a sensor array configured withsensors that generate event signals upon detection of a change inincident light intensity, and a display apparatus including a displaysection configured to change luminance of a planar regioninstantaneously with a predetermined spatial resolution as per acalibration pattern of the sensors.

According to another aspect of the present invention, there is provideda display control apparatus including a display control sectionconfigured to output an image signal to a display section configured tochange luminance of a planar region instantaneously with a predeterminedspatial resolution as per the image signal corresponding to acalibration pattern of sensors.

According to a further aspect of the present invention, there isprovided a program for causing a processing circuit connected with adisplay section to execute a process of outputting an image signal tothe display section configured to change luminance of a planar regioninstantaneously with a predetermined spatial resolution as per the imagesignal corresponding to a calibration pattern of sensors that generateevent signals upon detection of a change in incident light intensity.

According to an even further aspect of the present invention, there isprovided a sensor calibration method including the steps ofinstantaneously changing luminance of a planar region within spacecorresponding to an angle of view of a sensor array as per a calibrationpattern, and causing sensors constituting the sensor array to generateevent signals upon detection of a change in incident light intensity.

According to the above outlined configurations, a change in luminancethat occurs as a result of getting the display section to display acalibration pattern is detected in order to efficiently calibrate anevent-driven vision sensor.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting a schematic configuration of asensor calibration system implemented as a first embodiment of thepresent invention.

FIG. 2 is a view depicting a first example of calibration patterns.

FIG. 3 is a view depicting a second example of the calibration patterns.

FIG. 4 is a view depicting a third example of the calibration patterns.

FIG. 5 is a view depicting a fourth example of the calibration patterns.

DESCRIPTION OF EMBODIMENTS

Some preferred embodiments of the present invention are described belowin detail with reference to the accompanying drawings. Throughout theensuing description and the drawings, the constituent elements havingsubstantially identical functions and configurations are represented bythe same reference signs, and the redundant explanations are notrepeated.

FIG. 1 is a block diagram depicting a schematic configuration of asensor calibration system implemented as a first embodiment of thepresent invention. As depicted in FIG. 1, a calibration system 10includes a sensor apparatus 100 and a display apparatus 200. The sensorapparatus 100 includes an event-driven vision sensor 110 and a controlsection 120. The display apparatus 200 includes a display section 210and a display control section 220.

In the sensor apparatus 100, the vision sensor 110 includes a sensorarray 111 and a processing circuit 112 connected with each other, thesensor array 111 being configured with sensors 111A, 111B, etc.,corresponding to image pixels. The sensors 111A, 111B, etc., includelight receiving elements that generate event signals upon detectingchanges in incident light intensity, more particularly at the time ofdetecting a change in luminance. The event signals are output from theprocessing circuit 112 as information indicating timestamps, sensoridentification information (e.g., pixel positions), and luminance changepolarities (increase or decrease), for example. A subject moving withinan angle of view of the sensor array 111 changes the intensity of thelight being reflected or scattered. This makes it possible to detect amovement of the subject in chronological order using the event signalsgenerated by the sensors 111A, 111B, etc., corresponding to the edges ofthe subject, for example.

The control section 120 includes a communication interface 121, aprocessing circuit 122, and a memory 123. The communication interface121 receives the event signals sent from the processing circuit 112 ofthe vision sensor 110 and outputs the received event signals to theprocessing circuit 122. Further, the communication interface 121 maycommunicate with the display apparatus 200 over a wired or wirelesscommunication network. The processing circuit 122 processes the receivedevent signals by operating in keeping with a program stored typically inthe memory 123. For example, on the basis of the event signals, theprocessing circuit 122 generates in chronological order the imagesmapping the positions where luminance changes have occurred, stores thegenerated images in the memory 123 temporarily or permanently, andtransmits the images to another apparatus via the communicationinterface 121. As will be discussed later, the control section 120 mayanalyze the event signals based on calibration patterns.

In the display apparatus 200, the display section 210 is an apparatusconfigured to change the luminance of a planar region instantaneouslywith a predetermined spatial resolution, such as an LCD (Liquid CrystalDisplay), an OLED (Organic Light-Emitting Diode) display, or aprojector. In the ensuing description, the wording “changing luminancewith a predetermined spatial resolution” means dividing a given regionin space (e.g., planar region) into a predetermined number of regionsand changing the luminance of each of the divided regions. Further, thewording “changing luminance instantaneously” means causing luminancechanges in a short time by means of electronic switching. The LCD or theOLED display cited here as an example of the display section 210includes electronic light emitters such as backlights or natural lightemitters. In this case, the luminance of the emitted light can bechanged instantaneously. In other examples, a projector may be used asthe display section 210. The display section 210 is arranged in such amanner as to change the luminance of the planar region within the angleof view of the sensor array 111 at the time of performing calibration.The planar region of which the luminance is changed by the displaysection 210 corresponds to the display surface, for example. In the caseof the projector, the planar region of which the luminance is changed isa projection plane. In this case, the main body of the projectorconstituting the display section 210 may be located outside the angle ofview of the sensor array 111. As will be discussed later, the displaysection 210 is configured to change the luminance of the planar regionas per calibration patterns.

The display control section 220 includes a communication interface 221,a processing circuit 222, and a memory 223. The communication interface221 is configured to output image signals generated by the processingcircuit 222 to the display section 210. The communication interface 221may communicate with the sensor apparatus 100 over a wired or wirelesscommunication network. The processing circuit 222 is configured tooperate in keeping with a program stored typically in the memory 223 andthereby to generate the image signals corresponding to the calibrationpatterns displayed on the display section 210. The image signalscorresponding to the calibration patterns are output to the displaysection 210 via the communication interface 221. For example, theprocessing circuit 222 reads data indicative of the calibration patternsfrom the memory 223 or receives such data from another apparatus via thecommunication interface 221.

As mentioned above, with its ability to operate at high speed on lowpower, the event-driven vision sensor 110 is more advantageous than theframe-type vision sensor. The reason for this is that, of the sensors111A, 111B, etc., constituting the sensor array 111, only those thathave detected luminance changes generate event signals. Because thesensors not detecting luminance changes do not generate event signals,the processing circuit 112 can process and transmit at high speed onlythe event signals of those sensors having detected the luminancechanges. Since neither processing nor signal transmission takes place inthe case where there is no luminance change, it is possible for thesensor to operate on low power. On the other hand, even if the subjectis within the angle of view of the sensor array 111, there is noluminance change taking place unless the subject moves. This makes itdifficult to calibrate the vision sensor 110 when, for example,stationary calibration patterns are used as the subject.

In view of the above, the calibration system 10 calibrates the visionsensor 110 by causing the display section 210 to display calibrationpatterns. Specifically, the data representing the calibration patternsdisplayed on the display section 210 is transmitted from the displayapparatus 200 to the sensor apparatus 100, or an instruction to displaythe data indicative of a particular calibration pattern is transmittedfrom the sensor apparatus 100 to the display apparatus 200. This allowsthe processing circuit 122 of the sensor apparatus 100 to analyze theevent signals based on the calibration patterns. The sensors 111A, 111B,etc., constituting the sensor array 111 generate their event signals bydetecting a change in incident light intensity attributable to aluminance change in the planar region on the display section 210. Thismakes it possible to calibrate the vision sensor 110 by analysis of theevent signals based on the calibration patterns. For example, thedisplay section 210 may be caused to display the calibration patterns,which will be discussed below, to perform various types of calibrationon the vision sensor 110. Incidentally, the analysis of event signalsbased on calibration patterns may be carried out by the sensor apparatus100 as described above, by the display apparatus 200, or by anotherapparatus to which the data indicative of the calibration patterns andthe event signals are transmitted.

FIG. 2 is a view depicting a first example of calibration patterns forthe present embodiment. In this illustrated example, the display section210 displays a calibration pattern 211A and a calibration pattern 211Bin a switching manner. That is, the display section 210 changes theluminance of the display surface or of the planar region correspondingto the projection plane of a projector, first as per the calibrationpattern 211A and then as per the calibration pattern 211B. As a result,in the planar region, a spatial distribution of luminance as per thecalibration pattern 211A appears first, followed instantaneously by aspatial distribution of luminance as per the calibration pattern 211B.The calibration patterns 211A and 211B each include a high-luminanceregion 212 and a low-luminance region 213 arranged in a grid-likepattern. The high-luminance region 212 and the low-luminance region 213in the calibration pattern 211B are the inverse of their counterparts inthe calibration pattern 211A. Incidentally, in other examples, thehigh-luminance region 212 and the low-luminance region 213 are notlimited to the grid-like pattern and may be arranged in any desiredspatial pattern.

When the image displayed on the display section 210 of the abovecalibration system 10 is switched from the calibration pattern 211A tothe calibration pattern 211B, those of the sensors 111A, 111B, etc.,which correspond to the high-luminance region 212 in the calibrationpattern 211B generate event signals indicative of an increase inluminance. Likewise, those sensors that correspond to the low-luminanceregion 213 generate event signals indicative of a decrease in luminance.Comparing the positional relations of the sensors generating therespective event signals with the calibration patterns 211A and 211Bmakes it possible to detect, for example, an image distortion caused byan optical system (not depicted in FIG. 1) of the vision sensor 110 andthereby to correct internal and external parameters as well asdistortion factors defined in a manner similar to those of commoncameras.

FIG. 3 is a view depicting a second example of the calibration patternsfor the present embodiment. In this illustrated example, the displaysection 210 switches its display successively from a calibration pattern211C to a calibration pattern 211D to a calibration pattern 211E to acalibration pattern 211F. That is, the display section 210 causes theluminance of the display surface or of the planar region correspondingto the projection plane of the projection to change successively as perthe calibration patterns 211C through 211F. As a result, in the planarregion, spatial distributions of luminance as per the calibrationpatterns 211C through 211F appear successively, switching from onepattern to another instantaneously. The calibration patterns 211C, 211D,211E, and 211F are uniform in luminance and have luminance valuesdifferent from each other over the entire region of the display section210. The calibration pattern 211C has the lowest luminance value,followed by the calibration patterns 211D, 211E, and 211F with theirluminance values in ascending order, the calibration pattern 211F havingthe highest luminance value. Incidentally, although four luminancevalues are switched from one to another in the illustrated example, moreluminance values may be used and switched successively from one toanother. Whereas the calibration patterns when displayed are switchedfrom one to another in a monotonically increasing manner in theillustrated example, the calibration patterns may alternatively bedisplayed and switched from one to another in a monotonically decreasingmanner.

In the above calibration system 10, when the display section 210switches its image successively from the calibration pattern C throughto the calibration pattern F, those of the sensors 111A, 111B, etc.,which take the display section 210 for the subject generate eventsignals at some point during the switching. For example, in the casewhere the event signals are generated at the time of switching from thecalibration pattern 211D to the calibration pattern 211E, a thresholdvalue th at which the sensors detect a change in luminance is betweenthe luminance value of the calibration pattern 211D and that of thecalibration pattern 211E. If the threshold value thus identified failsto fall within a design range at production time of the vision sensor110, for example, the sensors 111A, 111B, etc., may be adjusted orreplaced.

FIG. 4 is a view depicting a third example of the calibration patternsfor the present embodiment. In this illustrated example, as in theexample explained above with reference to FIG. 2, the display section210 displays the calibration pattern 211C and the calibration pattern211F in a switching manner. Here, the calibration pattern 211C isconstituted with a uniform low-luminance region and the calibrationpattern 211F with a uniform high-luminance region.

In the above calibration system 10, when the display section 210switches its image from the calibration pattern 211C to the calibrationpattern 211F, theoretically all of the sensors 111A, 111B, etc., thattake the display section 210 for the subject generate event signalsindicative of an increase in luminance. Thus, any sensor that failed togenerate an even signal or has generated an event signal indicative of adecrease in luminance is identified as a defective pixel (luminescentspot) Likewise, when the display section 210 switches its image from thecalibration pattern 211F to the calibration pattern 211C, theoreticallyall of the sensors that take the display section 210 for the subjectgenerate event signals indicative of a decrease in luminance. At thispoint, any sensor that failed to generate an event signal or hasgenerated an event signal indicative of an increase in luminance is alsoidentified as a defective pixel (luminescent spot).

FIG. 5 is a view depicting a fourth example of the calibration patternsfor the present embodiment. In this illustrated example, as in theexample explained above with reference to FIG. 2, the display section210 displays the calibration pattern 211G and the calibration pattern211H in a switching manner. The calibration pattern 211G is fullyconstituted with the low-luminance region 213 except for a first portionof the planer region, specifically a top-left portion of the regionwhere there is a high-luminance region 212A as a luminance-invertedregion. On the other hand, the calibration pattern 211G is also fullyconstituted with the low-luminance region 213 except for a secondportion of the planer region different from the first portion thereof,specifically a bottom-right portion of the region where there is ahigh-luminance region 212B as the luminance-inverted region. In otherexamples, the luminance-inverted region is not limited in position tothe top-left corner or to the bottom-right corner, and the region may belocated anywhere. In a further example as the inverse of the illustratedexample, the calibration pattern may be fully constituted with thehigh-luminance region 212 except for a portion of the region made up ofthe low-luminance region 213 as the luminance-inverted region.

In the above calibration system 10, when the display section 210switches its image from the calibration pattern 211G to the calibrationpattern 211H, those sensors in the sensor array 111 that detect aluminance change in the first portion of the planar region generateevent signals indicating that a decrease in luminance has occurred as aresult of the high-luminance region 212A getting replaced with thelow-luminance region 213. On the other hand, those sensors in the sensorarray 111 that detect a luminance change in the second portion of theplanar region generate event signals indicating that an increase inluminance has occurred as a result of the low-luminance region 213getting replaced with the high-luminance region 212B. Except for a casewhere there exist effects of noise or defective pixels, the sensors thatdetect changes in luminance in the region other than the first portionor the second portion thereof do not generate event signals. Comparingthe timestamps of the event signals generated by the respective sensorsmentioned above makes it possible to compare the offset amounts of thetimestamps for each of the sensors in the case where only some of thesensors have generated event signals as a result of a luminance changesolely in a portion of the angle of view of the sensor array 111.

Although some preferred embodiments of the present invention have beendescribed in detail with reference to the accompanying drawings, theseembodiments are not limitative of this invention. It is obvious thatthose skilled in the art will easily conceive variations or alternativesof the embodiments within the scope of the technical idea stated in theappended claims. It is to be understood that such variations,alternatives, and other ramifications also fall within the technicalscope of the present invention.

REFERENCE SIGNS LIST

10: Calibration system

100: Sensor apparatus

110: Vision sensor

111: Sensor array

111A, 111B: Sensor

112: Processing circuit

120: Control section

121: Communication interface

122: Processing circuit

123: Memory

200: Display apparatus

210: Display section

220: Display control section

221: Communication interface

222: Processing circuit

223: Memory

211A to 211H: Calibration pattern

212: High-luminance region

213: Low-luminance region

1. A sensor calibration system comprising: a sensor apparatus includingan event-driven vision sensor including a sensor array configured withsensors that generate event signals upon detection of a change inincident light intensity; and a display apparatus including a displaysection configured to change luminance of a planar regioninstantaneously with a predetermined spatial resolution as per acalibration pattern of the sensors.
 2. The sensor calibration systemaccording to claim 1, wherein the planar region is within an angle ofview of the sensor array.
 3. The sensor calibration system according toclaim 1, wherein the display apparatus further includes a displaycontrol section configured to output an image signal corresponding tothe calibration pattern to the display section, and the sensor apparatusfurther includes a control section configured to analyze the eventsignals based on the calibration pattern.
 4. The sensor calibrationsystem according to claim 1, wherein the display section includes anelectronic light emitter.
 5. The sensor calibration system according toclaim 1, wherein the calibration pattern includes a first calibrationpattern and a second calibration pattern, and the display sectionchanges the luminance of the planar region as per the first calibrationpattern, and then changes the luminance of the planar region as per thesecond calibration pattern.
 6. The sensor calibration system accordingto claim 5, wherein the first calibration pattern includes ahigh-luminance region and a low-luminance region each arranged as per aspatial pattern, and the second calibration pattern includes thehigh-luminance region and the low-luminance region that are the inverseof their counterparts in the first calibration pattern.
 7. The sensorcalibration system according to claim 5, wherein the first calibrationpattern is constituted with a uniform low-luminance region, and thesecond calibration pattern is constituted with a uniform high-luminanceregion.
 8. The sensor calibration system according to claim 5, whereinthe first calibration pattern includes a luminance-inverted region in afirst portion of the planar region, and the second calibration patternincludes a luminance-inverted region in a second portion of the planarregion that is different from the first portion thereof.
 9. The sensorcalibration system according to claim 1, wherein the calibration patternincludes a plurality of calibration patterns having uniform luminancewith luminance values that differ from each other, and the displaysection successively changes the luminance of the planar region as perthe plurality of calibration patterns in such a manner that theluminance values either increase or decrease monotonically.
 10. Adisplay control apparatus comprising: a display control sectionconfigured to output an image signal to a display section configured tochange luminance of a planar region instantaneously with a predeterminedspatial resolution as per the image signal corresponding to acalibration pattern of sensors that generate event signals upondetection of a change in incident light intensity.
 11. The displaycontrol apparatus according to claim 10, wherein the display sectionincludes an electronic light emitter.
 12. The display control apparatusaccording to claim 10, wherein the calibration pattern includes a firstcalibration pattern and a second calibration pattern, and the imagesignal is output to the display section thereby causing the displaysection to change the luminance of the planar region as per the firstcalibration pattern, and then change the luminance of the planar regionas per the second calibration pattern.
 13. The display control apparatusaccording to claim 12, wherein the first calibration pattern includes ahigh-luminance region and a low-luminance region each arranged as per aspatial pattern, and the second calibration pattern includes thehigh-luminance region and the low-luminance region that are the inverseof their counterparts in the first calibration pattern.
 14. The displaycontrol apparatus according to claim 12, wherein the first calibrationpattern is constituted with a uniform low-luminance region, and thesecond calibration pattern is constituted with a uniform high-luminanceregion.
 15. The display control apparatus according to claim 12, whereinthe first calibration pattern includes a luminance-inverted region in afirst portion of the planar region, and the second calibration patternincludes a luminance-inverted region in a second portion of the planarregion that is different from the first portion thereof.
 16. The displaycontrol apparatus according to claim 10, wherein the calibration patternincludes a plurality of calibration patterns having uniform luminancewith luminance values that differ from each other, and the image signalis output to the display section thereby causing the display section tosuccessively change the luminance of the planar region as per theplurality of calibration patterns in such a manner that the luminancevalues either increase or decrease monotonically.
 17. A non-transitory,computer readable storage medium containing a program, which whenexecuted by a computer, causes a processing circuit connected with adisplay section to execute: outputting an image signal to the displaysection configured to change luminance of a planar regioninstantaneously with a predetermined spatial resolution as per the imagesignal corresponding to a calibration pattern of sensors that generateevent signals upon detection of a change in incident light intensity.18. A sensor calibration method comprising: instantaneously changingluminance of a planar region within an angle of view of a sensor arrayas per a calibration pattern; and causing sensors constituting thesensor array to generate event signals upon detection of a change inincident light intensity.